{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b21b469b",
   "metadata": {},
   "source": [
    "## 获取待提取特征的文件\n",
    "\n",
    "提供两种批量处理的模式：\n",
    "1. 目录模式，提取指定目录下的所有jpg文件的特征。\n",
    "2. 文件模式，待提取的数据存储在文件中，每行一个样本。\n",
    "\n",
    "当然也可以在最后自己指定手动提取指定若干文件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8dee942",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from glob import glob \n",
    "from onekey_algo import get_param_in_cwd\n",
    "\n",
    "mydir = r'E:\\OnekeyDS\\multi_modal_omics\\CT\\i_c4/*.nii.gz'\n",
    "samples = glob(mydir)\n",
    "samples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26847144",
   "metadata": {},
   "source": [
    "## 确定提取特征\n",
    "\n",
    "通过关键词获取要提取那一层的特征。\n",
    "\n",
    "### 支持的模型名称\n",
    "\n",
    "模型名称替换代码中的 `model_name`变量的值。\n",
    "\n",
    "| **模型系列** | **模型名称**                                                 |\n",
    "| ------------ | ------------------------------------------------------------ |\n",
    "| AlexNet      | alexnet                                                      |\n",
    "| VGG          | vgg11, vgg11_bn, vgg13, vgg13_bn, vgg16, vgg16_bn, vgg19_bn, vgg19 |\n",
    "| ResNet       | resnet18, resnet34, resnet50, resnet101, resnet152, resnext50_32x4d, resnext101_32x8d, wide_resnet50_2, wide_resnet101_2 |\n",
    "| DenseNet     | densenet121, densenet169, densenet201, densenet161           |\n",
    "| Inception    | googlenet, inception_v3                                      |\n",
    "| SqueezeNet   | squeezenet1_0, squeezenet1_1                                 |\n",
    "| ShuffleNetV2 | shufflenet_v2_x2_0, shufflenet_v2_x0_5, shufflenet_v2_x1_0, shufflenet_v2_x1_5 |\n",
    "| MobileNet    | mobilenet_v2, mobilenet_v3_large, mobilenet_v3_small         |\n",
    "| MNASNet      | mnasnet0_5, mnasnet0_75, mnasnet1_0, mnasnet1_3              |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e8d607a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from onekey_algo.custom.components.comp2 import init_from_onekey3d\n",
    "from monai.transforms import (\n",
    "    EnsureChannelFirst,\n",
    "    AddChannel,\n",
    "    Compose,\n",
    "    RandRotate90,\n",
    "    Resize,\n",
    "    ScaleIntensity,\n",
    "    EnsureType,\n",
    "    SqueezeDim,\n",
    ")\n",
    "\n",
    "model, transformer, device = init_from_onekey3d(r'C:\\Users\\onekey\\Desktop\\onekey_comp1\\comp4-What（分类识别）\\models3d\\ShuffleNet/viz')\n",
    "transformer = Compose([SqueezeDim(-2), EnsureChannelFirst(), ScaleIntensity(), Resize([96, 96, 96]), EnsureType()])\n",
    "for n, m in model.named_modules():\n",
    "    print('Feature name:', n, \"|| Module:\", m)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87f29370",
   "metadata": {},
   "source": [
    "## 提取特征\n",
    "\n",
    "`Feature name:` 之后的名称为要提取的特征名，例如`layer3.0.conv2`, 一般深度学习特征提取最后一层，例如`avgpool`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35e0d469",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from functools import partial\n",
    "from onekey_algo.custom.components.comp2 import extract3d, print_feature_hook, reg_hook_on_module\n",
    "from monai.data import ImageDataset\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "feature_name = r'avgpool'\n",
    "os.makedirs('features', exist_ok=True)\n",
    "with open('features/3DL_feature.csv', 'w') as outfile:\n",
    "    hook = partial(print_feature_hook, fp=outfile)\n",
    "    find_num = reg_hook_on_module(feature_name, model, hook)\n",
    "    val_ds = ImageDataset(image_files=samples, transform=transformer)\n",
    "    # create a validation data loader\n",
    "    val_loader = DataLoader(val_ds, batch_size=1, num_workers=0)\n",
    "    results = extract3d(val_loader, samples, model, device, fp=outfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d3cbd6f",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7ac1309",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "features = pd.read_csv('features/3DL_feature.csv', header=None)\n",
    "features.columns=['ID'] + list(f\"DL_{c}\" for c in features.columns[1:])\n",
    "features.to_csv('features/3DL_feature.csv', index=False)\n",
    "features.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9634e5f2",
   "metadata": {},
   "source": [
    "### 深度特征压缩\n",
    "\n",
    "深度学习特征压缩，注意压缩到的维度需要小于样本数\n",
    "\n",
    "```python\n",
    "def compress_df_feature(features: pd.DataFrame, dim: int, not_compress: Union[str, List[str]] = None,\n",
    "                        prefix='') -> pd.DataFrame:\n",
    "    \"\"\"\n",
    "    压缩深度学习特征\n",
    "    Args:\n",
    "        features: 特征DataFrame\n",
    "        dim: 需要压缩到的维度，此值需要小于样本数\n",
    "        not_compress: 不进行压缩的列。\n",
    "        prefix: 所有特征的前缀。\n",
    "\n",
    "    Returns:\n",
    "\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c649a10",
   "metadata": {},
   "outputs": [],
   "source": [
    "from onekey_algo.custom.components.comp1 import compress_df_feature\n",
    "\n",
    "cm_features = compress_df_feature(features=features, dim=64, prefix='DL_', not_compress='ID')\n",
    "cm_features.to_csv('features/3DL_compress_feature.csv', header=True, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1285696",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbd0d2e0",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
